ADP001293 


The  Maintenance  Performance  System 


Dr.  Douglas  J.  Bobko 
Research  Psychologist 
Army  Research  Institute 
Alexandria,  Virginia 


The  Maintenance  Performance  System  (MPS)  is  a  computer-based  manage¬ 
ment  system  designed  to  improve  the  conduct  and  quality  of  maintenance 
training  at  the  direct  support  level;  it  has  been  in  operation  at  a 
FORSCOM  maintenance  battalion  for  nearly  one  year.  An  evaluation  of  the 
system  suggests  that  MPS  monitors  daily  maintenance  activities  with 
relative  ease  and  accuracy.  Moreover,  MPS-generated  skill  and  training 
information  is  used  by  shop  personnel  to  diagnose  skill  deficiencies  and 
to  guide  technical  training  of  Individual  repairmen 
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Background 


The  proper  and  continuous  maintenance  of  Army  vehicles  and  other 
equipment  is  crucial  to  achieving  combat  readiness.  However,  a 
number  of  reports  have  indicated  that,  at  both  the  organizational  and 
direct  support  level,  equipment  maintenance  procedures  are  less  than 
optimal.  Typical  of  recurring  problems  in  maintenance  operations  are 
these:  faulty  diagnosis,  failure  to  recognize  equipment  failures, 
unnecessary  part  replacement,  and  failure  to  conduct  on-the-job  training. 

In  an  effort  to  help  correct  these  maintenance  deficiencies,  and 
with  the  assumption  that  effective  maintenance  training  helps  ensure 
efficient  maintenance  operations,  the  Army  Research  Institute  (ARI) 
developed  the  Maintenance  Performance  System  (MPS) . 

The  Maintenance  Performance  System 

The  primary  goal  of  MPS  is  to  improve  the  conduct  and  quality  of 
on-the-job  training  of  repairmen  in  their  performance  of  technical  tasks 
at  direct  support  battalions  (a  more  extensive  system  for  use  at  the 
organizational  level  is  now  being  designed).  To  accomplish  this,  MPS 
provides  unit  management  with  current  information  about  the  experience, 
training  and  skills  of  maintenance  personnel,  and  then  guides  the  manager 
toward  selection  of  available  training  methods  and  materials  to  meet 
skill  and  performance  needs. 

Integration  Into  Normal  Operations 

The  structure  of  MPS  and  its  integration  into  normal  operations  are 
shown  in  Figure  1.  Daily  changes  in  both  equipment  status  and  repairmen 
experience  are  recorded  and  fed  into  a  computer  file;  in  turn,  information 
about  shop  performance  on  an  aggregate  and  individual  level  is  prepared 
in  report  form.  These  reports  from  MPS  are  given  to  battalion  and 
company  level  managers  who  use  them  to  diagnose  maintenance  performance 
deficiencies.  Managers  can  then  use  specific  system-derived  guidance  to 
match  training  resources  with  training  needs  and  to  develop  an  on-the- 
job  training  program. 

It  is  important  to  note  that  MPS  differs  from  the  current  Array 
Maintenance  Control  System  (MCS).  MCS  is  primarily  designed  to  provide 
information  about  the  status  of  equipment  as  it  passes  through  the  shop 
and,  unlike  MPS,  MCS  provides  no  information  about  individual  and  unit 
skills,  experience,  or  training. 


Description  of  System 


MPS  has  been  in  use  for  nearly  one  year  by  two  forward  support  companies  of 
a  divisional  FORSCOM  maintenance  battalion.  It  currently  includes  ten  Military 
Occupational  Specialties  (MOS)  and  almost  all  of  the  equipment  serviced  by  the 


■.uuipaiues,  me  nuo s  mciuac  cne  nign  density  MOSs  of  63H  (track  vehicle 
repairmen),  63W  (wheel  vehicle  repairmen),  and  45K  (tank  turret  repairmen) 

Technical  MOS  supervisors  feed  information  into  the  system  through 
a  set  of  five  simple  input  forms.  Based  on  observations  to  date,  super¬ 
visors  spend  about  ten  minutes  each  week  completing  them.  The  information 
required  on  each  form  is  described  in  Table  1. 


TABLE  1.  DATA  COLLECTION  FORMS 


Title  Use 

Job  Order  Status  One  for  each  job;  records  time  of 

each  change  of  job  status 

Job  Performance  Records  amount  of  time  each 

repairman  works  on  each  job 

Daily  Man-Hour  For  each  repairman,  records 

Availability  available  hours,  direct  hours 

and  overtime 

Training/Performance  Records  accomplished  training 

Demonstration  or  testing  for  each  repairman 

Task  Experience  Repairman  entering  system  report 

prior  experience  on  each  task 


Day  to  day  operation  of  MPS  Is  accomplished  by  a  junior  enlisted 
clerk  in  grade  E3  or  E4.  The  clerk  is  responsible  for  collecting  the 
input  forms,  key  encoding  the  data  into  the  computer  (presently  an  IBM 
5120),  generating  computer-printer  reports,  and  distributing  the  reports. 
The  clerk  works  full  time  at  these  duties. 

In  all,  there  are  nine  different  MPS  reports  and  two  tables  (see 
Table  2).  One  table  lists  the  complete  roster  of  each  company  and  is 
distributed  to  the  company  office.  It  is  used  to  keep  track  of  the 
number  and  type  of  personnel  in  the  unit  on  a  bi-weekly  basis.  The 
other  table  contains  a  list  of  comments  about  local  conditions  that 
may  affect  the  interpretation  of  reports.  For  example,  this  table  would 
record  when  training  holidays  occur,  because  training  holidays  would 
affect  the  time  and  direct  labor  available  for  maintenance. 


Input 

Form 

1 

2 

3 

4 

5 


There  are  5  different  reports  which  contain  data  that  reflect  the  opera¬ 
tional  status  of  the  unit  and  these  are  delivered  to  relevant  personnel 
on  a  bi-weekly  schedule.  Maintenance  managers  use  information  about  man¬ 
hour  use,  time-to-repair,  and  time  in  each  job  status  (i.e.,  awaiting 
parts,  in-shop,  etc.)  to  locate  deficiencies  in  maintenance  performance 
and  other  problems.  Maintenance  managers  are  guided  in  the  diagnosis  of 
problems  by  an  "Interpretation  Guide"  that  is  specifically  keyed  to  MPS. 

Four  other  reports,  which  are  produced  every  six  weeks,  contain  data 
that  are  particularly  relevant  to  the  analysis  of  training  needs  and  the 
guidance  of  training  activities.  For  example,  the  Individual  Skill  History 
report  contains  a  record  of  each  individual's  experience  on  each  MOS  technical 
task.  Experience  on  a  task  Includes  the  job  exposure  that  occurs  when  a 
soldier  is  given  a  task  to  perform,  more  formal  training  that  is  provided 
to  train  or  refresh  skills,  and  demonstrated  proficiency  on  a  performance- 
based  test.  The  Individual  Skill  History  report  is  essentially  an  automated 
procedure  for  recording  the  tasks  that  each  soldier  performs  and  is  much 
like  a  job  book.  It  also  serves  as  the  basis  for  the  Training  Requirements 
Summary  report  which  actually  guides  training  activity.  This  report  liBts 
those  individuals  who  have  insufficient  experience  on  each  task  and  ranks 
the  need  for  training  according  to  the  degree  of  experience  and  the  crit¬ 
icality  of  the  task.  Training  managers  can  use  this  report  to  identify 
those  individuals  who  most  need,  training  on  each  task. 


TABLE  2.  MPS  REPORT  INFORMATION 


Table  1 
Table  2 
Report  1 
Report  2 
Report  3 

Report  4 
Report  5 
Report  6 
Report  7 
Report  8 
Report  9 


Roster 

Interpretation  Comments 

Man-hour  Availability  and  Use 

Average  Direct  Man-hours  Per  Job 

Average  Direct  Man-hours  Per  Job  by 
Equipment  and  Task 

Average  Job  Throughput  Time  in  Days 

Average  Days  Spent  in  Each  Job  Status 

Skill  and  Growth  Indices 

Skill  Development  Summary 

Individual  Skill  History 

Training  Requirements  Summary 
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evaluation  or  pit's 


MPS  has  been  in  operation  for  several  months  without  the  assistance 
of  or  supervision  by  ARI.  Several  visits  and  observations  during  this 
time  have  revealed  that  MPS  functions  smoothly  on  a  day-to-day  basis, 
e.g. ,  forms  are  routinely  attached  to  job  orders  and  filled  in,  data  is 
collected  and  entered  into  the  computer  regularly,  and  reports  jre 
printed  and  distributed  appropriately.  A  more  detailed  analysis  of  MPS 
was  conducted,  however,  to  evaluate  the  accuracy  and  validity  of  MPS 
information,  and  to  determine  the  extent  to  which  MPS  training  reports 
affect  the  conduct  and  content  of  skill  training  in  the  maintenance  shops. 

The  first  objective  of  this  evaluation  was  to  determine  the  accuracy 
of  information  being  entered  onto  the  more  critical  MPS  forms.  This  was 
done  in  light  of  the  fact  that  MPS  report  interpretation  and  consequent 
command  action  is  predicated  on  confidence  in  the  accuracy  of  MPS 
statistics;  data  input  quality  is  a  fundamental  problem  for  any  such 
system. 

A  second  objective  of  the  evalution  was  to  assess  the  validity  of 
MPS-generated  individual  skill  indices.  The  skill  index,  which  is  assigned 
to  each  repairman,  can  range  from  0  to  100  and  it  reflects  the  degree  to 
which  a  repairman  is  proficient  on  those  tasks  covered  by  his  MOS;  the 
skill  index  increases  each  time  a  repairman  works  on  a  job,  receives 
training,  or  passes  a  performance-based  test  on  a  job.  (The  skill  index 
is  essentially  a  single  number  describing  the  content  of  the  Individual 
.Skill  History  report. 

Skill  Indices  are  an  Integral  part  of  MPS  for  a  variety  of  reasons. 

For  battalion  and  company  commanders,  they  point  to  current  strengths 
and  weaknesses  in  overall  maintenance  capability,  and  provide  a  measure 
of  the  growth  in  maintenance  proficiency.  For  shop  supervisors,  the 
skill  index  is  an  easy  reference  to  those  repairmen  who  require  addi¬ 
tional  training.  To  the  individual  repairman  who  receives  a  skill 
growth  report  every  six  weeks,  the  skill  index  can  serve  as  a  source  of 
reward  and  motivation. 

The  third  objective  of  this  evaluation  was  to  determine  the  extent 
to  which  MPS  reports  of  job  experience  are  used  for  job  assignments  and 
for  setting  training  priorities. 

Accuracy  of  MPS  data 

Most  of  the  critical  information  monitored  by  MPS  is  collected  on 
MPS  Forms  1,  2,  and  3.  Hence,  a  sample  of  these  forms  in  each  company 
was  examined  and  checked  for  accuracy. 

MPS  Form  1  (Job  Order  Status).  For  two  companies  and  four  MOSs,  a 
total  of  67  forms  were  examined.  The  information  contained  on  these 
forms  was  compared  with  similar  information  contained  in  the  job  folder 
which  normally  follows  a  job  through  the  shop;  shop  officers  were  also 
questioned  about  data  entries  on  this  form.  It  was  found  that  the 
information  contained  on  MPS  Form  1  is  generally  consistent  with  that  of 
DA  Form  2407  and  with  verbal  reports  from  shop  officers.  The  most 
frequent  problem  was  an  incorrectly  entered  or  omitted  job  code;  this 
occurred  on  six  of  the  forms  examined. 
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selection,  these  forms  represented  shop  work  completed  within  the  last 
48  hours.  The  repairman  (and  their  supervisors)  listed  on  these  forms 
were  questioned  about  the  jobs  they  had  recently  performed,  and  the 
length  of  time  each  job  required.  In  each  case,  the  names  listed  on 
Form  2  were  consistent  with  reports  from  repairmen  and  supervisors, 
i.e.,  those  repairmen  who  performed  the  job  received  credit  for  it. 

The  accuracy  of  job  completion  time  data  was  more  difficult  to 
determine,  however.  Since  there  are  occasional  interruptions  in  work, 
and  because  start  and  stop  times  are  not  recorded  with  a  stopwatch,  job 
completion  times  are  generally  rough  estimates.  Based  on  reports  from 
repairmen  and  supervisors,  the  total  job  completion  time  is  typically 
rounded  up  to  the  nearest  hour  or  half  hour,  so  that  the  magnitude  of 
error  Increases  as  the  job  completion  time  decreases.  For  this  sample 
of  forms,  however,  there  was  no  instance  in  which  repairmen's  reports  of 
job  completion  time  differed  from  MPS  data  entries  by  more  than  60 
minutes. 

MPS  Form  3  (Dally  Man-Hour  Availability).  Fifty-five  forms  were 
reviewed,  which  represent  both  companies  and  four  MOSs.  From  interviews 
with  shop  suDervlsors,  it  is  clear  that  available  man-hour  entries  are 
accurate  but  that  the  direct  man-hour  entries  are  predominantly  inac¬ 
curate.  For  example,  it  is  a  common  practice  with  Form  3  to  enter  7 
or  3  direct  man-hours  for  repairmen  who  worked  only  4  or  5  hours. 

That  direct  man-hour  entries  are  inflated  was  confirmed  by  a  cross-check 
for  Form  3  with  Form  2  (which  records  job  performance  time  for  each 
repairman) . 

Validity  of  MPS  Skill  Index 

To  assess  the  validity  of  the  MPS  skill  index,  the  indices  were 
compared  with  independently  obtained  job  performance  ratings  of  each 
repairman.  In  short,  each  repairman  tracked  by  MPS  was  rated  by  both 
his  supervisors  and  his  peers.  Ratings  were  made  with  the  use  of  an 
anchored  scale,  where  a  0  performance  rating  indicated  a  repairman  who 
required  constant  and  total  supervision  on  a  job,  50  indicated  that  some 
supervision  was  required,  and  a  performance  rating  of  100  meant  that  the 
repairman  could  perform  a  task  successfully  without  any  supervision. 

Three  ratings  were  obtained  for  each  repairman:  the  first  rating  was  to 
indicate  a  repairman's  general  performance  in  his  MOS ,  and  the  remaining 
two  ratings  were  to  indicate  his  performance  on  two  different  but  speci¬ 
fic  tasks  (which  differed  for  each  MOS).  In  all,  usable  ratings  were 
obtained  from  20  supervisors  and  50  repairmen,  representing  four  MOSs 
(63H,  63G,  52D,  45K)  and  three  companies. 

Pearson  product-moment  correlations  were  calculated  to  determine 
the  relationship  between  MPS  skill  indices  and  performance  ratings;  for 
each  rater,  a  correlation  was  obtained  between  the  performance  ratings 
he  assigned  to  repairmen  and  those  repairmen's  3kill  indices.  While 
only  a  handful  (16%)  of  these  correlations  were  found  to  be  significant, 
the  majority  (86%)  of  them  were  positive.  In  all,  these  results  suggest 
a  small  yet  positive  relationship  between  MPS  skill  indices  and  performance 
ratings . 
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Effects  of  MPS  on  Training/Job  Assignment 


Structured  Interviews  were  conducted  with  shop  supervisors,  team 
leaders,  and  selected  repairmen  to  determine  what  effects  MPS  feedback 
has  on  maintenance  training,  job  assignment,  and  morale  or  motivation. 

Although  attitudes  toward  MPS  as  a  training  information  source  and 
guide  were  generally  positive,  the  actual  use  of  MPS  feedback  for  train¬ 
ing  purposes  was  varied  and  apparently  related  to  two  factors,  super¬ 
visory  level  and  MOS  density:  Warrant  officers,  for  example,  use  MPS 
feedback  more  often  than  team  leaders,  and  MPS  reports  play  a  greater 
role  in  larger  maintenance  sections  that  in  smaller  ones* 

MPS  skill  training  reports  were  used  for  training  purposes  in  the 
following  ways:  (1)  to  assign  jobs  to  work  teams,  i.e,  a  team  which  had 
done  some  job  least  often  would  be  assigned  that  job  over  other 
teams,  (2)  to  temporarily  shift  a  repairman  from  one  team  to  another  in 
order  to  gain  experience  on  some  job,  (3)  to  serve  as  a  memory  refresher 
about  which  repairmen  require  training  on  critical  skills,  and  (A)  to 
log  entries  into  job  books. 

Interviews  with  repairmen  revealed  that  individual  skill  history 
reports  had  little  effect  on  morale  or  motivation  for  two  reaons. 

First,  the  skill  history  reports  are  distributed  infrequently  (every  six 
weeks)  and  second,  the  reports  list  which  jobs  have  been  performed  but 
not  the  quality  of  performance. 


Discussion 

The  primary  purpose  of  MPS  is  to  Improve  the  conduct  and  quality  of 
on-the-job  maintenance  training.  Toward  this  end,  and  as  a  prototype 
system,  MPS  appears  to  have  worked  well.  It  has  been  shown  to  be  a 
system  in  which  dally  maintenance  activities  are  monitored  with  relative 
ease  and  accuracy.  Moreover,  MPS  provides  unique  maintenance  training 
information  and  guidance  which  is  used  for  training  purposes  by  shop 
personnel. 

The  results  of  the  evaluation  reported  here  and  other  observations 
made  during  the  past  year  suggest  several  improvements  or  modifications 
of  MPS.  For  example,  modification  of  the  MPS  computer  program  would 
allow  a  calculation  of  direct  man-hours  from  information  on  MPS  Form  2, 
thus  increasing  the  accuracy  and  reliability  of  the  man-hour  avail¬ 
ability  and  use  reports  (  and  eliminating  all  or  at  least  part  of  MPS 
Form  3).  Another  modification  would  be  to  increase  the  frequency  of 
distribution  of  individual  skill  history  reports,  and  hence  increase  the 
potential  this  feedback  has  for  motivation  and  morale.  One  other  important 
improvement  of  MPS  now  under  development  is  a  method  of  reflecting  the 
proficiency  with  which  tasks  are  performed;  currently,  two  repairmen 
will  receive  the  same  amount  of  credit  for  completing  a  task  regardless 
of  the  quality  of  their  work. 

ARI  is  now  working  toward  a  link  between  MPS  and  interactive  video¬ 
disc  instruction.  A  scenario  for  the  near  future  is  this:  After  signing 
on  at  a  computer  terminal,  a  mechanic  is  presented  with  a  summary  of  his 
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current  work  and  training  history.  The  computer  then  identifies  the 
mechanic's  training  deficiencies,  and  directs  him  toward  a  specific 
videodisc  lesson  available  for  use  on  that  terminal.  The  computer 
monitors  the  progress  of  the  mechanic  as  he  completes  the  lesson,  and 
updates  his  training  experience  accordingly. 

Finally,  it  should  be  noted  that  a  maintenance  system  like  MPS  on 
have  an  impact  on  more  than  the  unit  in  which  it  functions.  Specifically, 
maintenance  performance  information  can  be  fed  back  to  the  institutional 
training  base  for  use  in  curriculum  design,  and  it  can  provide  high 
quality  reference  data  to  the  equipment  design  process. 


